An actor system is the right shape for multi-model AI routing

Routing work across several model providers looks trivial until it isn't. Providers rate-limit you mid-burst. One goes down. Another is fast but weak on code; a third is expensive but the only one that clears the bar on a hard reasoning task. You want to prefer your own GPU nodes when their quality is good enough and fall back to a paid API only when it isn't. Do this with a pile of try/except and retry loops and you get a brittle knot nobody wants to touch.

Why the naive shape strains

A single synchronous request/response path has no natural place to hold state that outlives one call: which providers are healthy right now, how each has scored on this kind of task, what the current queue depth is. You end up bolting global mutable state onto stateless handlers and guarding it with locks. Backpressure (the thing that actually keeps you from melting a provider under a spike) has nowhere clean to live.

Mailboxes and supervision

Actors fit the problem because the problem is a lot of independent, stateful things that fail on their own. Each provider is an actor with its own mailbox and its own private state. A worker actor carries a capability registry (text-gen, code-gen, reasoning, tool-use) and posts metrics after every call. When an actor crashes, its supervisor restarts it without taking the system down. Messages queue in mailboxes, which is backpressure you get for free instead of backpressure you engineer.

The important shift: everything is a message. A request isn't a function call reaching into shared state: it's a message to a router, which sends a message to the best worker, which sends a message to a provider. State stays owned by one actor at a time, so there are no locks and no torn reads.

A smart router, not a load balancer

A load balancer spreads traffic evenly. That's the wrong objective. We want the cheapest model that clears the quality bar for this request. So the router scores candidates on a running quality mean per capability, checks provider health, and selects accordingly: local nodes first (sunk cost, near-zero marginal), premium APIs only when the local quality isn't sufficient. The spread between what a request is worth and what it costs to serve is the margin, and the router's whole job is to widen it.

The loop that compounds

Because every call posts back a quality signal, routing gets better with use. More traffic means a more accurate quality mean, which means better selection, which means higher acceptance and lower cost on the same hardware. Rejections aren't just losses: they're training signal for the next decision in that domain. The system's margins improve while you sleep, which is the opposite of a hand-tuned rule table that rots.

Cost is a first-class citizen

The reason this architecture matters commercially: it makes cost predictable and controllable. Priority queues keep critical traffic ahead of bulk work. Sensitivity-based routing sends regulated workloads to private infrastructure and everything else to the cheapest capable endpoint. You bring your own models and pay for orchestration, not for tokens you already own. An actor mesh is not the only way to build this, but it's the shape that stops fighting you the moment providers start behaving like the unreliable, heterogeneous things they actually are.